Comparing the return on investment of technologies to detect substandard and falsified amoxicillin: A Kenya case study

The prevalence of substandard and falsified medicines in low- and middle-income countries (LMICs) is a major global public health concern. Multiple screening technologies for post-market surveillance of medicine quality have been developed but there exists no clear guidance on which technology is optimal for LMICs. This study examined the return on investment (ROI) of implementing a select number of screening technologies for post-market surveillance of amoxicillin quality in a case study of Kenya. An agent-based model, Examining Screening Technologies using Economic Evaluations for Medicines (ESTEEM), was developed to estimate the costs, benefits, and ROI of implementing screening technologies for post-market surveillance of substandard and falsified amoxicillin for treatment of pediatric pneumonia in Kenya. The model simulated sampling, testing, and removal of substandard and falsified amoxicillin from the Kenyan market using five screening technologies: (1) Global Pharma Health Fund’s GPHF-Minilab, (2) high-performance liquid chromatography (HPLC), (3) near-infrared spectroscopy (NIR), (4) paper analytical devices / antibiotic paper analytical devices (PADs/aPADs), and (5) Raman spectroscopy. The study team analyzed the population impact of utilizing amoxicillin for the treatment of pneumonia in children under age five in Kenya. We found that the GPHF-Minilab, NIR, and PADs/aPADs were similar in their abilities to rapidly screen for and remove substandard and falsified amoxicillin from the Kenyan market resulting in a higher ROI compared to HPLC. NIR and PADs/aPADs yielded the highest ROI at $21 (90% Uncertainty Range (UR) $5-$51) each, followed by GPHF-Minilab ($16, 90%UR $4 - $38), Raman ($9, 90%UR $2 - $21), and HPLC ($3, 90%UR $0 - $7). This study highlights screening technologies that can be used to reduce costs, speed up the removal of poor-quality medicines, and consequently improve health and economic outcomes in LMICs. National medicine regulatory authorities should adopt these fast, reliable, and low-cost screening technologies to better detect substandard and falsified medicines, reserving HPLC for confirmatory tests.


Introduction
utilizing PADs/aPADs to screen for substandard and falsified amoxicillin in Kenya [14]. However, to our knowledge, no study has analyzed the return on investment comparing screening technologies in LMICs. This study estimates the return on investment of implementing HPLC, PADs/aPADs, GPHF-Minilab, NIR, and TruScan Raman spectroscopy in surveillance of antibiotic quality in Kenya. The results of this analysis are aimed at informing decision-making for national regulatory bodies and may serve as an example for other LMICs on optimal post-market surveillance strategies.

Methods
This is a return-on-investment study. The study team developed an agent-based model, Examining Screening Technologies using Economic Evaluations for Medicines (ESTEEM), to estimate the costs, benefits, and return on investment of implementing screening technologies for post-market surveillance of substandard and falsified amoxicillin in Kenya. The ESTEEM model has been piloted and validated in a previous study, where detailed methods can be found [14]. A literature search was conducted on existing technologies used to detect poorquality medicines in LMICs. Five technologies that are relevant in LMICs were identified: PADs/aPADs, GPHF-Minilab, NIR, TruScan Raman spectroscopy, and HPLC [10][11][12]17].
The ESTEEM model simulated sampling, testing, and removal of substandard and falsified medicines from the Kenyan market using the four screening technologies and HPLC. The agent-drug in the model represents amoxicillin treatments used to manage pneumonia among children under age five in Kenya [18,19]. The flow of agent-drug through the simulation model is illustrated in Fig 1. Based on a previous study, 13 brands of amoxicillin were identified to be circulating on the Kenyan market [20]. Two brands were identified as having some substandard and falsified products ( Table 1) [20]. Agents (amoxicillin treatments) were assigned a brand and given the characteristic of being either substandard/falsified or standard quality based on this market data.
The model simulates sampling of medicines over a three-month time period during which 40 samples of 100 pills are selected at random from each of the 13 brands (520 samples in total) [20,27]. Screening of all the samples using a screening technology occurs based on the time per device stated in Table 2. After the screening, confirmatory testing is simulated to take one month during which time all samples that failed screening plus 10% of those samples not  failing the screening undergo HPLC. Confirmed failed samples then undergo full pharmacopeial testing, taking 3 months. Finally, all batches linked to the failed samples are identified by the batch number and removed from the market. Implementing this process takes 9 months in total for PADs/aPADs, Raman, and NIR, 10 months for GPHF Minilab, and 19 months for HPLC on its own. This was run over a three-year period, where the sampling and testing with each technology occurred three times, while the scenario of HPLC on its own occurred only once.
To estimate the health and economic benefits of each screening scenario, we tracked the effects of using different screening devices for poor-quality antibiotics in terms of their impact on pediatric pneumonia outcomes. For each month of the model, the prevalence of substandard and falsified antibiotics was recorded and applied to the monthly cases of pediatric pneumonia. Pneumonia cases that received substandard/falsified versions of amoxicillin experienced double the case fatality rate and longer duration of illness. Detailed methods are described in the previous work [14].  The costs of implementing each screening technology as well as the costs of pediatric pneumonia cases are tracked throughout the model. The base cost of each screening device was added only once. Costs of treatment and productivity losses were calculated based on careseeking, duration of treatment, and deaths due to pneumonia [21][22][23]26]. A discount rate of 3% was applied on future productivity losses.
Six scenarios were simulated. The base case scenario simulated no intervention, allowing the natural flow of poor-quality medicines through the market. Four scenarios simulated the use of screening technologies (PADs/aPADs, GPHF-Minilab, Raman, and NIR) prior to HPLC confirmation. The final scenario simulated the use of HPLC on its own to test every sample. These five scenarios were compared to the base case scenario to calculate outcomes. The primary outputs of the model were the costs of implementing each scenario and the benefits resulting from its implementation in terms of substandard and falsified treatments removed from the market, deaths averted, and costs including productivity losses averted. The costs per benefit were calculated as well as the return on investment for each screening scenario.
A probabilistic sensitivity analysis was conducted to capture uncertainties around our estimates. Gamma distributions were assumed for cost inputs, and beta distributions for probabilities and prevalence of substandard and falsified medicines. This paper presents averages over 10,000 model runs with the 90% uncertainty range (UR), a range that captures 90% of the simulated outputs.

Results
The monthly prevalence of substandard and falsified medicines over three years for each scenario is compared in Fig 2. In the base case scenario, no poor-quality amoxicillin was removed from the market, resulting in a stable amount of substandard and falsified amoxicillin available to patients, as indicated by the straight line in Fig 2. Testing amoxicillin products using HPLC took the longest time, which meant that substandard and falsified amoxicillin was left to circulate until the removal period at month 19. Consequently, by the time of removal many of the batches identified as substandard/falsified were already utilized by patients or had expired and left the market. For the HPLC scenario, an annual average of 3,188 (90%UR 1,642-4,819) substandard and falsified amoxicillin treatments were removed, while all other scenarios removed more than double that amount ( Table 3).
Of the four screening technologies, PADs with aPADs were the most effective at identifying poor-quality amoxicillin, as demonstrated by the steeper drop in months where removal takes place (Fig 2). The PADs/aPADs scenario saw the highest annual removal of poor-quality amoxicillin treatments ( Table 3, n = 11,613, 90%UR 7,777-15,627). Similarly, GPHF-Minilab, NIR, and Raman were all effective at identifying substandard and falsified amoxicillin. However, due to the longer duration of sample testing with GPHF-Minilab, the removal of medicines in that scenario occurred a month later than the other three screening devices (Fig 2). Increases in prevalence seen between removal months were the result of the natural influx of substandard and falsified medicines in the market as new batches were added each month and old ones expired (Fig 2).
The total cost of implementing post-market surveillance with a screening technology included costs for sampling, testing, and then removal of amoxicillin. Annual sampling costs were estimated at $11,965 (90%UR $7,489 -$17,107) for three months of sampling in each of the three years for PADs/aPADs, NIR, Minilab, and Raman. For HPLC, only one three-month sampling period was included, resulting in a three-year cost of sampling of $11,965. Costs of personnel to remove substandard/falsified amoxicillin were estimated at $997 (90%UR $624 -$1,425) annually across the four portable screening technologies. Total annual cost to implement these technologies was lowest for NIR at $ 16 Table 3).
The estimated average number of substandard/ falsified treatments and the number of deaths averted with the implementation of various screening technologies and HPLC is presented in Table 3. Without any screening measures in place, on average 28,364 (90%UR  In the base case scenario, in which substandard and falsified amoxicillin was left to circulate on the market freely, annual direct treatment costs were estimated at approximately $13 million (90%UR $4.5 -$30 million) for childhood pneumonia ( Table 3). Short-and long-term productivity losses of pediatric pneumonia were estimated at $5 million (90%UR $4.3 -$6.8 million) and $315 million (90%UR $228 -$414 million), respectively. Implementing PADs/ aPADs, GPHF-Minilab, NIR, or Raman was estimated to save between $11.4 million to $15.2 million in direct treatment costs and averted productivity losses annually. All screening devices and HPLC were found to have a high return on investment. The highest returns on investment were associated with utilization of NIR at $21 (90%UR $5 -$51) and PADs/aPADs also at $21 (90%UR $6 -$51) per dollar invested, owing to the low cost of both devices. For every dollar invested, we estimated an expected return of $3 (90%UR $0 -$6) for HPLC, $9 for Raman (90%UR $2 -$21), and $16 (90%UR $4 -$38) for GPHF-Minilab ( Table 3). HPLC and Raman were estimated to have similar incremental costs, but Raman exhibited increasing benefits over time due to its ability to identify and subsequently remove substandard and falsified amoxicillin much faster (Fig 3).

Discussion
The health and economic impact of different screening technologies was estimated in the context of post-market surveillance of amoxicillin in Kenya. The study results indicate that PADs/ aPADs, GPHF-Minilab, NIR, and Raman are very similar in their abilities to quickly identify substandard and falsified amoxicillin and can have a substantial impact when used to narrow down which poor-quality medicines to remove from the market. We show that speeding up testing using any screening device results in greater returns than using HPLC on its own. Fewer substandard and falsified amoxicillin treatments were simulated to be administered to children under age five for pneumonia when implementing screening before HPLC, resulting in fewer annual deaths. Overall, all screening technologies resulted in a positive return on investment indicating that interventions to screen for substandard and falsified products are a worthwhile economic investment.
Despite their similarities, these screening technologies vary in terms of device acquisition costs, personnel training needs, testing costs, and time requirements. Fast, reliable, and lowcost drug screening tools are essential for increasing the detection of substandard and falsified medicines in LMICs [5,31]. The NIR is a field-portable device that uses dispersive spectroscopy to identify the API without destroying the sample or requiring any sample preparation [12,31]. In this study, the NIR yielded a higher ROI than using only HPLC owing to the lower device cost and much lower testing costs. A previous study compared the implementation of six screening devices in Laos, finding that with a high prevalence of substandard and falsified antimalarials, implementing any device was cost-effective, with NIR found to be most cost-effective [32]. The present analysis adds to the existing literature, finding NIR to have one of the highest ROIs in a different LMIC setting and for a different medicine class.
Raman and GPHF-Minilab screening technologies also yielded higher ROI than using only HPLC to screen amoxicillin samples. The Truscan Raman spectrometer is a nondestructive field-portable device that employs a dispersive spectroscopy technique analogous to the NIR system [11,17]. The acquisition of the Truscan Raman may seem cost-prohibitive for some regulatory authorities in LMICs, as they are about as expensive as the initial investment in HPLC. However, it can be utilized as an alternative to HPLC for expedited screening of medicine quality, and according to our simulations would be quicker than using HPLC to screen all samples. The GPHF-Minilab, deemed as a 'lab-in-a-suitcase', is a field-portable device employing disintegration and semi-quantitative thin layer chromatography (TLC) technique that has been extensively and successfully used to detect substandard and falsified medicines in LMICs [12,17,33,34]. This study demonstrated that the GPHF-Minilab has a relatively high ROI, further supporting its continued use for post-market surveillance in LMICs.
PADs and aPADs are newer, inexpensive paper-based colorimetric test cards that have been developed to screen for poor-quality antibiotics [20,[28][29][30]. In our simulations, PADs/ aPADs identified the most substandard and falsified antibiotics. This is due to the high sensitivity and specificity of utilizing PADs along with the associated antibiotic-specific aPADs which can be used to detect substandard amoxicillin [20]. A recent study demonstrated that utilization of PADs/aPADs for screening poor-quality amoxicillin in Kenya resulted in faster removal of substandard and falsified products and higher annual incremental benefits compared to using HPLC alone [14].
Costing results should be utilized in conjunction with a broader understanding of the environment where these screening devices will be implemented. For example, one study found that users wrongly categorized antimalarial samples more often when using PADs compared to using other screening devices [35]. We chose to evaluate screening devices that would be specifically suited to use in LMICs in that they do not require electricity or a special laboratory setup. Some other important factors to consider are the training level of the personnel conducting testing, the speed of the tests, and whether quantitative (level of API) or qualitative results (existence of specific API) are needed for the evaluation [10]. The underlying prevalence of falsified compared to substandard medicines can also be an important consideration, as all screening devices have been found to be less cost-effective in scenarios of low prevalence [32]. Using a combination of these screening devices could also be an effective strategy. A recent study conducted in Ghana demonstrated the usefulness of combining three low cost (<USD $10,000) screening methods (GPHF-Minilab, colorimetry, and Counterfeit Drug Indicator) to quickly detect substandard and falsified artemether/lumefantrine products [36].
Utilizing HPLC on its own yielded the lowest ROI compared to the four screening technologies, despite being able to accurately identify all substandard or falsified medicines. This is owing to the amount of time that HPLC requires to test individual samples. During that time, poor-quality batches remain on the market where they are likely to be used by patients. Despite the differences between these screening technologies, the model estimates that doing some post-market surveillance, even with the most expensive options, still yields a return on investment by averting deaths and costly hospital care.
HPLC is regarded as the gold standard for analyzing API. Most pharmacopeias, including the United States, European, and British Pharmacopeias, utilize it for compendial analysis of pharmaceutical products to establish acceptable quality standards [13,37]. However, given the high costs and resources needed to conduct HPLC, a stepwise three-level approach has been proposed for medicine quality control in LMICs. This involves (1) visual/physical inspection first, followed by (2) simple, rapid, and cost-effective screening tests such as TLC and spectroscopic technologies, and finally (3) confirmatory pharmacopeial analysis (e.g. HPLC) [38]. This analysis provides evidence for the return on investment of strengthening this second level of quality control involving screening technologies. Kenya has two WHO prequalified Quality Control Laboratories with HPLC capability for confirmatory pharmacopeial tests: the Ministry of Health's National Quality Control Laboratory (NQCL) and Laboratory of the Mission for Essential Drugs and Supplies (MEDS) [39]. Incorporating screening technologies can considerably reduce the burden of compendial testing in these laboratories and the cost burden on the regulatory authority.
Sustainable funding of national medicine regulatory agencies is crucial for strengthening and expanding current post-market surveillance capability in Africa [40]. While national regulatory agencies largely bear the costs of implementing post-market surveillance activities, the net benefits accrued have far-reaching health and economic impacts [40]. This study should serve as a call to action for national medicine regulatory authorities to shift current post-market surveillance practices from initial HPLC analysis to the stepwise three-level approach. Particularly, implementation of initial field screening using simple, reliable, and fast technologies can be used to expedite post-market surveillance and lower its total cost [15]. For successful adoption of these technologies, national governments should prioritize passing country-specific legislation to facilitate swift and appropriate response to medicines failing screening tests [41]. Further research is still needed to fill the gaps in the literature on the capabilities, costs, and benefits of screening technologies to inform post-market surveillance decisions in more LMICs.
This study has some key limitations. First, there was limited reliable and quality data for Kenya. The ESTEEM model fails to capture the geographical heterogeneity of Kenya in that assumptions were made for a variety of inputs based on local or regional data. For instance, for brand market share, data from a study in Western Kenya was used [20]. An estimate for the case fatality rate of childhood pneumonia deaths for children under age five across Africa was employed [24,25]. It was also assumed that personnel testing amoxicillin products across the country have the same level of training, and therefore, the specificity and sensitivity of devices would be relatively uniform. The study team utilized the most recent reliable data sources available at this time by conducting a thorough literature search. Second, in estimating the ROI of using a select screening and testing device, it was assumed that once a substandard and falsified medication is confirmed with HPLC, the associated batch was removed from the market before patient consumption. Third, this paper primarily focused on a single use of amoxicillin for the treatment of pneumonia among children without considering its other indications. Therefore, the cost and benefit outcomes are largely conservative. Furthermore, while this was a case study in Kenya, the results might vary based on country-specific regulatory and quality control capabilities, ultimately affecting the selection of screening technology. The type of medicine tested using a specific screening technology also affects the sensitivity, specificity, cost, and overall impact to be realized. Further analysis with more inclusive data across countries is warranted.

Conclusions
This study highlights screening technologies with a higher ROI than the current HPLC standard for medicine quality control. While HPLC largely remains the standard instrument for confirmatory testing, other technologies can be used to reduce costs and speed up the removal of poor-quality medicines, and consequently improve health and economic outcomes. Regulatory authorities should use this data to strengthen their surveillance efforts. This will enable respective government ministries of health to save costs both directly and indirectly through improved health from the consumption of quality medicines.